UNITED STATES
                       SECURITIES AND EXCHANGE COMMISSION
                             Washington, D.C. 20549

                                  SCHEDULE 13D
                  Under the Securities and Exchange Act of 1934
                              (Amendment No. ___)*

                         Casual Male Retail Group, Inc.
- --------------------------------------------------------------------------------
                                (Name of Issuer)

                     Common Stock, $0.01 par value per share
- --------------------------------------------------------------------------------
                         (Title of Class of Securities)

                                    148711104
                         -------------------------------
                                 (CUSIP Number)

                         Chilton Investment Company, LLC
                        1266 East Main Street, 7th Floor
                               Stamford, CT 06902
                                 (203) 352-4000
- --------------------------------------------------------------------------------
(Name, Address and Telephone Number of Person Authorized to Receive Notices and
                                 Communications)

                                   May 5, 2006
- --------------------------------------------------------------------------------
             (Date of Event which Requires Filing of this Statement)

If the filing person has previously filed a statement on Schedule 13G to report
the acquisition that is the subject of this Schedule 13D, and is filing this
schedule because of Rule 13d-1(e), Rule 13d-1(f) or Rule 13d-1(g), check the
following box |X|.

Note: Schedules filed in paper format shall include a signed original and five
copies of the schedule, including all exhibits. See Rule 13d-7 for other parties
to whom copies are to be sent.

*The remainder of this cover page shall be filled out for a reporting person's
initial filing on this form with respect to the subject class of securities, and
for any subsequent amendment containing information which would alter
disclosures provided in a prior cover page.

The information required on the remainder of this cover page shall not be deemed
to be "filed" for the purpose of Section 18 of the Securities Exchange Act of
1934 or otherwise subject to the liabilities of that section of the Act but
shall be subject to all other provisions of the Act (however, see the Notes).



- ----------------------------                           -------------------------
CUSIP No. 148711104                SCHEDULE 13D        Page 2 of 6 Pages
- ----------------------------                           -------------------------


- --------------------------------------------------------------------------------
1    NAME OF REPORTING PERSON
     I.R.S. IDENTIFICATION NO. OF ABOVE PERSON (ENTITIES ONLY)

     Chilton Investment Company, LLC
- --------------------------------------------------------------------------------
2    CHECK THE APPROPRIATE BOX IF A MEMBER OF A GROUP  (a) |_| (b) |X|

- --------------------------------------------------------------------------------
3    SEC USE ONLY

- --------------------------------------------------------------------------------
4    SOURCE OF FUNDS

     AF, OO
- --------------------------------------------------------------------------------
5    CHECK IF DISCLOSURE OF LEGAL PROCEEDINGS IS REQUIRED PURSUANT TO ITEMS
     2(d) or 2(e) |_|

- --------------------------------------------------------------------------------
6    CITIZENSHIP OR PLACE OF ORGANIZATION

     Delaware
- --------------------------------------------------------------------------------
                7    SOLE VOTING POWER

                     7,147,945
   NUMBER OF    ----------------------------------------------------------------
    SHARES      8    SHARED VOTING POWER
 BENEFICIALLY
   OWNED BY          0
     EACH       ----------------------------------------------------------------
   REPORTING    9    SOLE DISPOSITIVE POWER
    PERSON
     WITH            7,147,945
                ----------------------------------------------------------------
                10   SHARED DISPOSITIVE POWER

- --------------------------------------------------------------------------------
11     AGGREGATE AMOUNT BENEFICIALLY OWNED BY EACH REPORTING PERSON

       7,147,945
- --------------------------------------------------------------------------------
12     CHECK IF THE AGGREGATE AMOUNT IN ROW (11) EXCLUDES CERTAIN SHARES |_|

- --------------------------------------------------------------------------------
13     PERCENT OF CLASS REPRESENTED BY AMOUNT IN ROW (11)

       20.0%
- --------------------------------------------------------------------------------
14     TYPE OF REPORTING PERSON (See Instructions)

       IA
- --------------------------------------------------------------------------------



- ----------------------------                           -------------------------
CUSIP No. 148711104                SCHEDULE 13D        Page 3 of 6 Pages
- ----------------------------                           -------------------------

ITEM 1.     Security and Issuer.

This statement on Schedule 13D (the "Statement") relates to the common stock,
par value $0.01 per share (the "Common Stock") of Casual Male Retail Group,
Inc., a Delaware corporation (the "Issuer" or the "Company"). The principal
executive offices of the Issuer are located at 555 Turnpike Street, Canton, MA
02021. This Statement supersedes the statement on Schedule 13G filed by the
Reporting Person on February 14, 2006 (the "Schedule 13G").

ITEM 2.     Identity and Background.

This Statement is being filed by Chilton Investment Company, LLC, a limited
liability company organized in the state of Delaware (sometimes referred to
herein as "Chilton" or the "Reporting Person").

Chilton is the general partner of certain investment limited partnerships (the
"Chilton Funds") and the investment adviser of certain managed accounts (the
"Managed Accounts" and, together with the Chilton Funds, the "Client Accounts"),
certain of which own Common Stock of the Issuer. As such, Chilton exercises
voting control and dispositive power over such Common Stock. Chilton's Client
Accounts are primarily engaged in the business of investing and trading in
securities.

The address of the principal office of the Reporting Person is 1266 East Main
Street, 7th Floor, Stamford, Connecticut 06902.

During the past five years, the Reporting Person (i) has not been convicted in a
criminal proceeding (excluding traffic violations or similar misdemeanors) and
(ii) has not been a party to a civil proceeding of a judicial or administrative
body of competent jurisdiction which resulted in a judgment, decree or final
order enjoining future violations of, or prohibiting or mandating activities
subject to, Federal or State securities laws or finding any violation with
respect to such laws.

ITEM 3.     Source and Amount of Funds or Other Consideration.

As of the date hereof, Chilton has acquired on behalf of its Client Accounts
7,147,945 shares of Common Stock (including 697,133 shares that can be acquired
by the Client Accounts pursuant to immediately convertible 5% convertible senior
subordinated noted due 2024) for an aggregate purchase price of $57,951,802.58.
All of such purchases and sales were made through open market transactions. The
funds for such acquisitions came from the working capital of the Client Accounts
or from their margin accounts in the regular course of business.

ITEM 4.     Purpose of Transaction.

The investment in the Common Stock by Chilton, on behalf of its Client Accounts,
was made in the ordinary course of its business of investing and trading in
securities, as reported on the Schedule 13G. Chilton acquired the Common Stock
believing such securities to be undervalued.



- ----------------------------                           -------------------------
CUSIP No. 148711104                SCHEDULE 13D        Page 4 of 6 Pages
- ----------------------------                           -------------------------

Chilton intends to continue to review on a regular basis its investment in the
Common Stock and the Company's business and affairs, as well as the price level
of the Common Stock, conditions in the securities markets and general economic
and industry conditions. Chilton may in the future take such actions with
respect to its investment in the Common Stock as it deems appropriate in light
of the circumstances existing from time to time, including acquiring or
disposing of Common Stock at any time and from time to time in the open market
or otherwise, and communicating its views regarding the Company to third
parties.

ITEM 5.     Interest in Securities of the Issuer.

(a) As of today, the Reporting Person is deemed to beneficially own (through its
control of the Client Accounts) 7,147,995 shares of Common Stock (including
697,133 shares that can be acquired by the Client Accounts pursuant to
immediately convertible 5% convertible senior subordinated noted due 2024)
representing 20.0% of the Issuer's Common Stock. The foregoing calculations are
based on 35,038,632 shares of Common Stock issued and outstanding as of March
29, 2006 according to publicly available filings of the Issuer, together with
697,133 shares that can be acquired by the Client Accounts pursuant to
immediately convertible 5% convertible senior subordinated notes due 2024, for a
total of 35,735,765.

(b) The responses of the Reporting Person to Items 7 through 11 of the cover
pages of this Schedule 13D relating to beneficial ownership of the shares of
Common Stock are incorporated herein by reference.

(c) The following transactions in shares of Common Stock were effected during
the preceding 60 days by the Reporting Person, in each case on behalf of one or
more Client Accounts:

o     On March 8, 2006, 16,927 shares of Common Stock were purchased through
      Goldman Sachs & Co. at a price of $8.5341 per share.
o     On March 16, 2006, 200,000 shares of Common Stock were purchased through
      Weeden Securities at a price per share of $9.7225.
o     On March 17, 2006, 58,200 shares of Common Stock were purchased through
      Weeden Securities at a price per share of $9.6979.
o     On March 17, 2006, 157,600 shares of Common Stock were purchased through
      Liquinet at a price per share of $9.7988.
o     On March 17, 2006, 84,200 shares of Common Stock were purchased through
      Morgan Stanley & Co. at a price per share of $9.8938.
o     On March 22, 2006, 7,028 shares of Common Stock were sold through Morgan
      Stanley & Co. at a price per share of $9.4972.
o     On March 23, 2006, 200,000 shares of Common Stock were sold through Knight
      Securities at a price per share of $10.0583
o     On March 24, 2006, 100,000 shares of Common Stock were sold through Knight
      Securities at a price pr share of $10.0475.
o     On March 24, 2006, 100,000 shares of Common Stock were sold through
      Liquinet at a price per share of $10.0425.




- ----------------------------                           -------------------------
CUSIP No. 148711104                SCHEDULE 13D        Page 5 of 6 Pages
- ----------------------------                           -------------------------

o     On March 24, 2006 38,664 shares of Common Stock were sold through B-Trade
      at a price per share of $10.0046.
o     On March 27, 2006, 20,000 shares of Common Stock were sold through Merrill
      Lynch & Company at a price per share of $9.9784.
o     On March 27, 2006, 41,336 shares of Common Stock were sold through B-Trade
      at a price per share of $9.9185.
o     On March 29, 2006, 6,840 shares of Common Stock were purchased through
      Goldman Sachs & Company at a price per share of $9.8373.
o     On March 31, 2006, 56,600 shares of Common Stock were purchased through
      Jeffries & Company at a price per share of $9.4068.
o     On March 31, 2006, 43,474 shares of Common Stock were purchased through
      B-Trade at a price per share of $9.6973
o     On April 3, 2006, 12,400 shares of Common Stock were purchased through
      Credit Suisse First Boston Securities at a price per share of $9.5779.
o     On April 3, 2006, 45,254 shares of Common Stock were purchased through
      B-Trade at a price per share of $9.5477.
o     On April 3, 2006, 100,000 shares of Common Stock were purchased through
      Jeffries & Company at a price per share of $9.5583.
o     On April 3, 2006, 115,000 shares of Common Stock were purchased through
      Liquinet at a price per share of $9.5843.
o     On April 4, 2006, 23,400 shares of Common Stock were purchased through
      B-Trade at a price per share of $9.6697.
o     On April 4, 2006, 16,300 shares of Common Stock were purchased through
      Jeffries & Company at a price per share of $9.7531.
o     On April 5, 2006, 52,530 shares of Common Stock were purchased through
      B-Trade at a price per share of $9.8109.
o     On April 5, 2006, 17,730 shares of Common Stock were sold through B-Trade
      at a price per share of $9.74.
o     On April 5, 2006, 2,110 shares of Common Stock were purchased through
      Goldman Sachs Company at a price per share of $9.8482.
o     On April 6, 2006, 35,042 shares of Common Stock were purchased through
      Credit Suisse First Boston Securities at a price per share of $9.7884.
o     On April 19, 2006, 1,930 shares of Common Stock were sold through Morgan
      Stanley & Company at a price per share of $9.4244.
o     On April 28, 2006, 25,000 shares of Common Stock were purchased through
      Jeffries & Company at a price per share of $9.3480.
o     On May 1, 2006, 22,649 shares of Common Stock were purchased through
      Jeffries & Company at a price per share of $9.4661.
o     On May 3, 2006, 1,531 shares of Common Stock were purchased through Morgan
      Stanley & Company at a price per share of $9.3875.
o     On May 4, 2006, 3,300 shares of Common Stock were purchased through
      B-Trade at a price per share of $9.4991.
o     On May 4, 2006 296,700 shares of Common Stock were purchased through
      Liquinet at a price per shares of $9.3699.
o     On May 4, 2006, 150,000 shares of Common Stock were purchased through
      Liquinet at a price per share of $9.35.



- ----------------------------                           -------------------------
CUSIP No. 148711104                SCHEDULE 13D        Page 6 of 6 Pages
- ----------------------------                           -------------------------

o     On May 4, 2006, 50,000 shares of Common Stock were purchased through
      Canter Fitzgerald at a price per share of $9.3700.
o     On May 4, 2006, 12,842 shares of Common Stock were purchased through
      B-Trade at a price per share of $9.4913.
o     On May 4, 2006, 119,900 shares of Common Stock were purchased through
      Weeden Securities at a price per share of $9.3400.
o     On May 4, 2006, 167,258 shares of Common Stock were purchased through
      Jeffries & Co. at a price per share of $9.5500.
o     On May 4, 2006, 20,176 shares of Common Stock were purchased through Banc
      of America at a price per share of $9.7303.
o     On May 4, 2006, 223,700 shares of Common Stock were purchased through
      Jeffries & Co. at a price pre share of $9.8897.
o     On May 4, 2006, 170,398 shares of Common Stock were purchase through Banc
      of America at a price per share of $10.4739.
o     On May 4, 2006, 55,902 shares of Common Stock were purchased through
      B-Trade at a price per share of $10.5097.
o     On May 4, 2006, 50,000 shares of Common Stock were purchased through
      Weeden Securities at a price per share of $10.6000.

(d) Chilton's Client Accounts that own the Common Stock have the right to
receive dividends from, and the proceeds from any sale of, such Common Stock.

(e) Not applicable.

ITEM 6.     Contracts, Arrangements, Understandings or Relationships with
            Respect to Securities of the Issuer.

Not applicable.

ITEM 7.     Material to Be Filed as Exhibits.

Not applicable.

SIGNATURE

After reasonable inquiry and to the best of my knowledge and belief, I certify
that the information set forth in this statement is true, complete and correct.

Date:       May 9, 2006

                                    Chilton Investment Company, LLC

                                    By:/s/ James Steinthal
                                       -----------------------------------------
                                       James Steinthal
                                       Managing Director