Use the file command. It will either tell you 'mc68020 demand paged' if it is an a.out binary, and give a longer (self-explanatory) description if it is ELF.
You can also use ldd. If it says something about a "DLL jump", the binary is in a.out format; otherwise, it's in ELF.
For statically linked binaries, ldd reports statically linked (ELF) for ELF binaries. I have no idea what it says for a.out binaries, because I don't have anything statically linked a.out any more (at least, not that I can find).