I think the main reason independents vote against the Party of the President in the first midterm election, is that the new President has actually done some things their opposition can complain about. The complaints are not necessary valid, but there is some truth in the complaints in that they usually are about some things the new President has done. Independents have no political positions, so they don't really think about those things being good or bad. Somebody in a suit on TV is complaining about them, so that's all the input they need!
Right back to the founding fathers, they worried about an uneducated mob following demagogues down the garden path. Most of the Independents ARE that mob. A bunch of poorly-informed, easily-manipulated people determining who will run the country. Tea-partiers complained about "the elite", but "elite" in this case actually means people who are educated and have some awareness of the consequences of idiots running everything.
Trump said: "I love the uneducated," for a reason.