An auction is a process which allows multiple publishers to bid on your work. Auctions can vary in shape and type, depending on the number of editors interested, the type of book, and the agent involved.
Before an auction occurs, an agent will get in touch with editors and explain the rules. They will set a date in which offers from publishers must be received and what terms must be included.
An auction is typically classified as either rolling or best bid. In a rolling auction, after all initial offers are received the lowest bidder is giving an opportunity to improve on the highest offer currently on the table. The next lowest editor can then try to beat this, and so on, until every editor who made an intitial bid has had an opportunity to re-bid. Those who don't are dropped, and a new round begins, starting again with the editor with the lowest offer on the table. This process can continue without limit until editors give their top bid, or an agent can establish beforehand a closing date to ensure that the auction doesn't get drawn out indefinitely. A best bid option is simpler - the editor must convey to the agent their best bid immediately, without knowing what other editors are bidding.
As I mentioned in a previous post, some publishers might try to pre-empt an auction by conveying an offer for a limited period. If the author and agent reject this pre-empt, this can sometimes be adapted as a floor offer - i.e. the minimum offer that a bid has to be for in an auction. By making a floor offer a publisher then gets the right to top the last bid made by ten percent if such auction occurs.
Your agent should establish clear rules for the auction, guided by what they believe to be the best strategy for a successful sale. No matter what, these rules should state that the author reserves the right to accept an offer on the basis of a publisher's comprehensive proposal rather than based on the highest dollar offer, since the most money isn't always the best deal.