Land use change is a pressing concern for the livelihoods of people in tropical developing countries. Changes in land use from swidden (shifting cultivation) agriculture to smallholder tree‐dominated areas producing timber, fruits and cash crops can result in changing livelihood outcomes for rural communities. This paper examines land use patterns of rural households and the association with food production and income across three different zones of various forest proximity across a landscape gradient (remote, intermediate and on‐road) in the Chittagong Hill Tracts region of Bangladesh. We conducted in‐depth semi‐structured surveys of households (175–300) and farm owners (30) to collect information on people’s perceptions of land use change, present land use patterns and contributions to food production and income. Our research found that more than half of the surveyed households experienced a decline in the land available for food production over the past 30 years. The land use patterns revealed decreasing crop lands (mainly swidden shifting cultivation/land rotation farms) and an increase in areas of planted trees within this landscape. However, household use of the reduced crop land has not affected food production in the on‐road zone, whereas the diversity of food sources has declined. People living in more remote areas engaged in swidden farming and used larger areas of crop and fallow lands, fruit orchard and accessed natural forest lands that provide a diverse reservoir of food sources. The current land uses contribute to variations in annual household income across zones, with remote dwelling people earning less to those living closer to urban areas in the intermediate and on‐road zones. In summary, this transition of land uses over three decades and changes in income and food availability cannot be generalised across the region because of zone specific differences. We recommend a broader and context‐reliant landscape management approach in consideration of the diversity of forest and tree benefits for the livelihoods of people in the region.