Because your example sentence uses the word ‘went’ rather than ‘was’, you need a comma because those are two separate I dependent clauses.

You and Dave were together and then Dave leaves you and goes driving by himself… me and Dave, then Dave went.

If you used ‘was’ then those would not be independent clauses and therefore a comma would not be used. It was me and Dave and Dave was driving.