If you drop the vibration slider in the options to zero, does that make a difference?
What about if you switch to windowed or fake fullscreen display mode?
Maybe try uninstalling the drivers, reboot and see if the game still recognizes the controllers (the drivers may only be needed for force feedback). If so, does that affect input for player 1? Disconnecting and reconnecting should automatically reinstall the drivers.
Do you have a different controller you can try as a test?
Try using
x360ce to configure the 2 controllers slightly differently.
Run x360ce_x64.exe from the install folder ('shipping' subfolder), configure both controllers (manually, or searching for the "most popular settings for my controller"), and in the Game Settings set it to create Xinput files for 64 bit v1.3. Hopefully that will avoid the problem.