SECURITIES AND EXCHANGE COMMISSION
                             Washington, D.C. 20549
                                  ------------

                                    FORM 11-K

                                  ANNUAL REPORT
                        PURSUANT TO SECTION 15(D) OF THE
                         SECURITIES EXCHANGE ACT OF 1934


        [X]  ANNUAL REPORT PURSUANT TO SECTION 15(d) OF THE SECURITIES
             EXCHANGE ACT OF 1934

        FOR THE FISCAL YEAR ENDED DECEMBER 31, 2003

        OR

        [ ]  TRANSITION REPORT PURSUANT TO SECTION 15(d) OF THE SECURITIES
             EXCHANGE ACT OF 1934
        For the transition period from                     to
                                        ------------------    ------------------


                         Commission File Number 1-13154


A. Full title of the plan and the address of the plan, if different from that of
   the issuer named below:

                            AMERICAN MEDICAL SECURITY
                             RETIREMENT SAVINGS PLAN


B. Name of issuer of the securities held pursuant to the plan and the address of
   its principal executive office:

                      AMERICAN MEDICAL SECURITY GROUP, INC.
                               3100 AMS Boulevard
                               Green Bay, WI 54313




                American Medical Security Retirement Savings Plan

                           Annual Report on Form 11-K
                   For the Fiscal Year Ended December 31, 2003



                                TABLE OF CONTENTS

                                                                            PAGE
Financial Statements and Supplemental Schedule
Years ended December 31, 2003 and 2002

         Report of Independent Registered Public Accounting Firm...............3

         Financial Statements

         Statements of Net Assets Available for Benefits.......................4
         Statements of Changes in Net Assets Available for Benefits............5
         Notes to Financial Statements.........................................6

         Supplemental Schedule

         Schedule H, Line 4i--Schedule of Assets (Held at End of Year)........10

Signatures....................................................................11

Exhibit Index...............................................................EX-1

                                       2





             Report of Independent Registered Public Accounting Firm

The Administrative Committee
American Medical Security
Retirement Savings Plan

We have audited the accompanying statements of net assets available for benefits
of American Medical Security Retirement Savings Plan as of December 31, 2003 and
2002, and the related statements of changes in net assets available for benefits
for the years then ended. These financial statements are the responsibility of
the Plan's management. Our responsibility is to express an opinion on these
financial statements based on our audits.

We conducted our audits in accordance with the standards of the Public Company
Accounting Oversight Board (United States). Those standards require that we plan
and perform the audit to obtain reasonable assurance about whether the financial
statements are free of material misstatement. An audit includes examining, on a
test basis, evidence supporting the amounts and disclosures in the financial
statements. An audit also includes assessing the accounting principles used and
significant estimates made by management, as well as evaluating the overall
financial statement presentation. We believe that our audits provide a
reasonable basis for our opinion.

In our opinion, the financial statements referred to above present fairly, in
all material respects, the net assets available for benefits of the Plan at
December 31, 2003 and 2002, and the changes in its net assets available for
benefits for the years then ended, in conformity with U.S. generally accepted
accounting principles.

Our audits were performed for the purpose of forming an opinion on the financial
statements taken as a whole. The accompanying supplemental schedule of assets
(held at end of year) as of December 31, 2003, is presented for purposes of
additional analysis and is not a required part of the financial statements but
is supplementary information required by the Department of Labor's Rules and
Regulations for Reporting and Disclosure under the Employee Retirement Income
Security Act of 1974. This supplemental schedule is the responsibility of the
Plan's management. The supplemental schedule has been subjected to the auditing
procedures applied in our audits of the financial statements and, in our
opinion, is fairly stated in all material respects in relation to the financial
statements taken as a whole.

                                           /s/ Ernst & Young LLP


April 2, 2004
Milwaukee, Wisconsin

                                       3





                AMERICAN MEDICAL SECURITY RETIREMENT SAVINGS PLAN

                 Statements of Net Assets Available for Benefits



                                                                                   DECEMBER 31
                                                                             2003              2002
                                                                      -------------------------------------
ASSETS
                                                                                        
Investments, at fair value (NOTE 3):
   Mutual funds                                                             $54,428,654       $40,565,278
   Common stock of American Medical Security
     Group, Inc.                                                                731,061           612,651
   Participant loans receivable                                               1,086,347         1,050,515
   Profit sharing receivable from American Medical
     Security, Inc.                                                                   -         1,269,383
                                                                      =====================================
Net assets available for benefits                                           $56,246,062       $43,497,827
                                                                      =====================================



                                       4



                AMERICAN MEDICAL SECURITY RETIREMENT SAVINGS PLAN

           Statements of Changes in Net Assets Available for Benefits



                                                                             YEAR ENDED DECEMBER 31
                                                                            2003               2002
                                                                     --------------------------------------
                                                                                       
Investment gain (loss):
   Net realized and unrealized appreciation
    (depreciation) in fair value of investments (NOTE 3)                   $10,282,420       $ (6,847,272)
   Interest and dividends                                                      621,365            318,617
                                                                     --------------------------------------
                                                                            10,903,785         (6,528,655)
   Less administrative expense                                                   5,500             79,623
                                                                     --------------------------------------
                                                                            10,898,285         (6,608,278)
Contributions:
   Employers'                                                                2,237,034          3,740,059
   Employees'                                                                4,409,024          4,783,013
                                                                     --------------------------------------
                                                                             6,646,058          8,523,072
   Other                                                                        67,478             10,855
                                                                     --------------------------------------
Total additions                                                             17,611,821          1,925,649

Benefits paid                                                                4,863,586          3,613,923
                                                                     --------------------------------------
Net increase (decrease)                                                     12,748,235         (1,688,274)

Net assets available for benefits at beginning of year                      43,497,827         45,186,101
                                                                     --------------------------------------
Net assets available for benefits at end of year                           $56,246,062        $43,497,827
                                                                     ======================================


                                       5




                American Medical Security Retirement Savings Plan

                          Notes to Financial Statements

                                December 31, 2003


1.  DESCRIPTION OF THE PLAN

The following description of the American Medical Security Retirement Savings
Plan (the Plan) provides only general information. Participants should refer to
the Plan document for a complete description of the Plan's provisions.

GENERAL

The Plan is a contributory defined contribution plan covering all full-time and
part-time employees of American Medical Security Group, Inc. (AMSG) and its
subsidiaries, American Medical Security, Inc. and Nurse Healthline, Inc.
(collectively the Company or Employer). The Plan is subject to the provisions of
the Employee Retirement Income Security Act of 1974 (ERISA).

PARTICIPATION AND VESTING

Employees are eligible to become contributing participants in the Plan and
receive matching contributions on the first day of the following month after
reaching the age of 18 and after 30 days of eligible service. Employees with 30
days of service that are employed on the last day of the Plan year are entitled
to receive profit-sharing contributions for the Plan year.

Participants are immediately vested in their contributions plus actual earnings
thereon. Participants do not vest in Company contributions until three years of
service, at which time they become fully vested.

The nonvested portion of the Company's profit sharing and matching contributions
are forfeited when a participant terminates employment. Profit sharing
forfeitures are allocated to the remaining Plan participants, while matching
contribution forfeitures are used to reduce future matching contributions.
During 2003 and 2002, the Company utilized forfeited amounts of $179,000 and
$65,000, respectively, to offset the required matching contribution.

                                       6





                American Medical Security Retirement Savings Plan

                    Notes to Financial Statements (continued)


1.  DESCRIPTION OF THE PLAN (CONTINUED)

CONTRIBUTIONS, WITHDRAWALS AND LOANS

Plan participants are permitted to make contributions on a before-tax basis each
payroll period of 2% to 18% of base compensation, subject to the maximum amount
allowed by the Internal Revenue Code (IRC). Participants can change their
before-tax contribution percentage any time during the Plan year. Participants
direct current contributions and accumulated contributions and earnings between
funds offered through the Plan.

In 2003 and 2002, the Company matched 70% of the first 6% of compensation
contributed to the Plan by a participant. Additional amounts may be contributed
at the option of the Company. In 2002, the Company made a discretionary profit
sharing contribution of $1,269,000, or 2% of eligible wages. No discretionary
profit sharing contribution was made in 2003.

Distributions due to retirement, death, permanent disability and termination of
employment are provided for as defined within the Plan.

Participants may borrow from their fund accounts a minimum of $1,000 up to a
maximum of the lesser of one-half of their vested account balance or $50,000.
The loan is secured by the balance in the participant's account and bears
interest at a rate commensurate with prevailing rates as determined by the Plan
administrator. Principal and interest are repaid through payroll deductions.

TERMINATION OF THE PLAN

The Company has established the Plan with the intention and expectation that the
Employer will be able to make contributions indefinitely, but the Employer
neither is, nor shall, be under any obligation or liability whatsoever to
maintain the Plan for any given length of time. The Company retains the right to
modify or terminate the Plan at any time, but may not retroactively reduce the
share of any participant or cause the Plan's assets to revert to the Company
unless required by law. In the event of termination, the balance of each
participant's account would become fully vested, and all assets would be
distributed to the participants and beneficiaries.

                                       7



               American Medical Security Retirement Savings Plan

                    Notes to Financial Statements (continued)


1.  DESCRIPTION OF THE PLAN (CONTINUED)

EXPENSES

Administrative expenses of the Plan are paid from Plan assets to the extent not
paid by the Company.

2.  SUMMARY OF SIGNIFICANT ACCOUNTING POLICIES

INVESTMENT VALUATION AND ACCOUNTING METHOD

The Plan's financial statements are prepared on the accrual basis of accounting.
The Plan's investments are stated at fair value. The fair value of mutual fund
shares and AMSG common stock are based on the quoted market values on the last
business day of the Plan year. The fair value of the participation units in the
common trust fund is based on quoted redemption values. Participant loans are
stated at their unpaid principal balance, which approximates fair value.

USE OF ESTIMATES

The preparation of financial statements in conformity with U.S. generally
accepted accounting principles requires the Company to make estimates that
affect the amounts reported in the Plan's financial statements and accompanying
notes. Actual results could differ from these estimates.

3.  INVESTMENTS

The following individual investments represent 5% or more of the Plan's net
assets available for benefits:

                                                             DECEMBER 31
                                                        2003              2002
                                                   -----------------------------
Investments at fair value as determined by quoted
market price or redemption value:
     Alleghany Montague & Caldwell Growth Fund     $11,612,658        $8,968,682
     MCM Stable Asset Fund                           6,774,529         6,335,953
     Scudder Growth & Income Fund                    5,508,156         4,217,761
     CS Warburg Pincus Emerging Growth Fund          4,762,419         3,274,196
     Merrill Lynch International Value Fund          4,458,953         2,894,452
     Neuberger Berman Genesis Trust Fund             5,411,797         4,020,858
     SSgA Emerging Markets Fund                      3,121,175         1,756,953
     Gabelli Equity Income Fund                      3,617,008         3,373,587


                                       8



               American Medical Security Retirement Savings Plan

                    Notes to Financial Statements (continued)


3.  INVESTMENTS (CONTINUED)

The Plan's investments, including gains and losses on investments bought, sold
and held during the year, appreciated (depreciated) in fair value as follows:

                                                      YEAR ENDED DECEMBER 31
                                                       2003             2002
                                                  ------------------------------

Mutual funds                                      $  9,907,242      $(6,903,851)
Common stock of American Medical
  Security Group, Inc.                                 375,178           56,579
                                                  ------------------------------
                                                  $ 10,282,420      $(6,847,272)
                                                  ==============================

4.  INCOME TAX STATUS

The underlying non-standardized prototype plan has received an opinion letter
from the Internal revenue Service (IRS) stating that the form of the plan is
qualified under Section 401 of the Internal Revenue code, and therefore, the
related trust is tax exempt. In accordance with the Revenue Procedure 2002-6 and
Announcement 2001-77, the Plan Sponsor has determined that it is eligible to and
has chosen to rely on the current IRS prototype plan opinion letter. Once
qualified, the Plan is required to operate in conformity with the Code to
maintain its qualification. The plan administrator believes the Plan is being
operated in compliance with the applicable requirements of the Code and,
therefore, believes that the Plan is qualified and the related trust is tax
exempt.

5.  TRANSACTIONS WITH PARTIES IN INTEREST

Certain administrative expenses are provided by the Company at no cost to the
Plan.

                                       9




                American Medical Security Retirement Savings Plan

                    Employer Identification Number 39-1431799
                                 Plan number 001

         Schedule H, Line 4i - Schedule of Assets (Held at End of Year)

                                December 31, 2003



                     Identity of Issuer, Borrower/                          UNITS/            CURRENT
                       DESCRIPTION OF INVESTMENT                            SHARES             VALUE
-----------------------------------------------------------------------------------------------------------

                                                                                     
American Century / International Growth Investors Fund                         34,127      $     270,967
American Funds Washington Mutual Fund                                          60,821          1,744,352
Dreyfus GNMA Fund                                                              90,317          1,342,110
Dreyfus S&P 500 Fund                                                           38,842          1,259,651
Gabelli Equity Income Fund                                                    220,549          3,617,008
Merrill Lynch International Value Fund                                        199,238          4,458,953
Alleghany Montague & Caldwell Growth Fund                                     532,935         11,612,658
MCM Stable Asset Fund                                                         493,782          6,774,529
INVESCO Small Company Growth Fund                                             130,466          1,450,781
Janus Twenty Fund                                                              17,629            637,624
Neuberger Berman Genesis Trust Fund                                           146,146          5,411,797
Strong Corporate Bond Fund                                                    121,583          1,287,561
Scudder Growth & Income Fund                                                  273,493          5,508,156
CS Warburg Pincus Emerging Growth Fund                                        173,811          4,762,419
CS Warburg Pincus Global Fixed Income Fund                                    115,734          1,168,913
SSgA Emerging Markets Fund                                                    185,674          3,121,175
American Medical Security Group, Inc. common stock*                            59,388            731,061
Participant loans receivable (interest rates range
  from 5.00% to 9.75%)                                                                         1,086,347
                                                                                        -------------------
                                                                                             $56,246,062
                                                                                        ===================


*Represents a party in interest to the Plan.

                                       10





                                   SIGNATURES


         Pursuant to the requirements of the Securities Exchange Act of 1934,
the Administrative Committee of the American Medical Security Retirement Savings
Plan has duly caused this annual report to be signed on its behalf by the
undersigned hereunto duly authorized.


DATE:             June 25, 2004


                                            AMERICAN MEDICAL SECURITY
                                            RETIREMENT SAVINGS PLAN

                                            /S/ JOHN R. LOMBARDI
                                            ------------------------------------
                                            John R. Lombardi
                                            American Medical Security Retirement
                                            Savings Plan Administrative
                                            Committee Member


                                       11




                                  EXHIBIT INDEX
                                       TO
                             FORM 11-K ANNUAL REPORT
                     for fiscal year ended December 31, 2003

EXHIBIT      DESCRIPTION                       INCORPORATED HEREIN        FILED
NUMBER                                           BY REFERENCE TO        HEREWITH

  23         Consent of Independent Registered
             Public Accounting Firm                                         X


                                      EX-1