Do I have to change my sex at the same notary's office where my birth certificate is kept?

No. Article 118 of Law 1395 of 2010 established that "All acts, facts and orders that must be registered in the civil registry or that affect it, may be registered in any office authorised to carry out the civil registry function in the national territory or in the Colombian consulates abroad".