Welcome, if your new A has a generator, what you observed is common. Not much you can do about it, other than changing to an alternator.
I think I would disagree. Maybe a change in brightness, but that would be a battery issue. Going completely black means that the lights are being lighted by something other than the battery, perhaps the generator under the influence of a defective voltage regulator?
I'll reference to the best Model A Circuit diagram in the world (there are others.) Any investigation you do will be referencing this diagram. It is the place you start in diagnosis.
https://lesmodela.files.wordpress.com/2017/10/wiring-diagram.pdf
Now looking at this one keeps in mind a 12V conversion doesn't change much except the voltage. UNLESS those who converted it did something seriously different than this diagram - which is always possible.
Referencing the diagram, keep in mind the "yellow" wire is the one which "feeds" your lights and your horn from the generator cut-out. Also see that a further yellow/black wire passes from cut-out to the junction box connection/pole. This is your generator connection to the car/battery.
See also that the ammeter connects together the two "poles" of the junction box. For all practical purposes an ammeter is a "closed circuit" meaning no resistance. The two poles are connected together electrically with the bonus feature you can read how much current passes between poles.
See also the additional yellow wire passing from the "other" junction pole and connecting to the black wire from the battery. Thus Ammeter is a way to see how much current flows to the battery, or from the battery to your car.
Somewhere along this path is likely your issue with "connection." You may be running only on generator voltage which is describing the symptoms you see. You don't say if your generator is "regulated" which I might expect as an open circuit Model A generator will generate as much as 32 volts which will burn out your light bulbs. (It is possible that your generator is an uprated Model A generator. This is possible with the correct voltage regulator. Although many retro-fit with a modern genny simply on the basis of reliability - which for most is the point of a 12V conversion anyway.)
Check each connection in turn. Use (my favorite) No-Al-Ox which is a conductive grease that electricians used to use on aluminum wire back when the code allowed you to use it. The grease contains graphite which itself is conductive and will keep and maintain any electrical connection. Thus you can "be sure" each connection is made as best as it can be made, and the grease seals it and keeps it this way.
Good luck. Diagnosis/solutions to problems that crop up is part of the attraction to many of the hobby - and what sets the Model A apart is that it is possible to do this yourself.
Joe K