システムリプレースの中でCDCを取り入れている話/chugokudb32-cdc

Transcript

1. γεςϜϦϓϨʔεͷதͰ
   CDCΛऔΓೖΕͯΔ࿩
   2023-10-07
   தࠃ஍ํDBษڧձ
   ೔ຊPostgreSQLϢʔβʔձ
   ߴڮ Ұٍ
   

   View Slide

2. ࣗݾ঺հ
   • ߴڮ Ұٍ (@ikkitang)
   • ελʔϑΣεςΟόϧגࣜձࣾ
   Tech Product Manager
   • TypeScriptͱGo
   • ೔ຊPostgreSQLϢʔβʔձ
   தࠃࢧ෦௕
   

   View Slide

3. ͓͠ͳ͕͖
   • CDC࠾༻ͷഎܠ
   • CDCͱ͸ʁ
   • Debeziumͷಛ௃
   

   View Slide

4. ஫ҙࣄ߲
   • εϥΠυ͸ެ։ࡁΈͰ͢
   • ࣭໰ͱ͔͋Ε͹ #ChugokuDB Λ͝׆༻͍ͩ͘͞ʂ
   • ࿩ͷ࣠ͱͯ͠"γεςϜϦϓϨʔε"͕ग़ͯ͘ΔͷͰ
   ՝୊ײͷڞײͷҙਤΛࠐΊͯ
   ॴଐͷձࣾͷ࿩Λଟ͘͠·͕͢࠾༻໨త͡Όͳ͍Ͱ͢
   • ※ CDC࠾༻ͷഎܠ͕݁ߏϘϦϡʔϜΛ͠Ίͯ͠·͍·ͨ͠
   

   View Slide

5. CDC࠾༻ͷഎܠ
   

   View Slide

6. CDC࠾༻ͷഎܠ
   ελʔϑΣεςΟόϧגࣜձࣾ
   • 2009೥7݄7೔૑ۀʢ15ظ໨ʣ
   • BtoB(C) ޲͚ͷϏδωεϞσϧΛల։
   • ҿ৯ళ༷ͷECࢀೖɾӡ༻Λτʔλϧࢧ
   ԉ
   • ECαΠτ্Ͱ஫จɾܾࡁड෇
   • ੡଄঎඼ͷडऔ
   • ੡଄঎඼ͷ͓ಧ͚
   

   View Slide

7. CDC࠾༻ͷഎܠ
   ελʔϑΣεςΟόϧגࣜձࣾͷϓϩμΫτͷ՝୊ײ
   • 3छྨͷΞΫλʔ
   • ੡଄ύʔτφʔ, ഑ૹύʔτφʔ, ٤৯ऀ༷
   • 15ظͷϏδωεΛࢧ͑ͨෳࡶͳۀ຿ϑϩʔʢి࿩ʣ
   • ͦΕΒΛશ෦ࢧ͑Δ "தԝूݖͷΞϓϦέʔγϣϯ"
   • ʮγεςϜͷ౎߹Ͱ...͝ڠྗΛ͓ئ͍͍ͨ͠·͢ʯ
   

   View Slide

8. CDC࠾༻ͷഎܠ
   γεςϜϦϓϨʔεΛ΍Δ
   • γεςϜϦϑΝΫλϦϯάͰ͸ͳͯ͘ɺϦϓϨʔε.
   • ϦϑΝΫλϦϯά͸֎෦తͳৼΔ෣͍Λม͑ͳ͍ࣄ͕ఆٛ
   • ৼΔ෣͍΋ม͑ͳ͍ͱɺ՝୊ղܾʹͳΒͳ͍ͷͰม͍͑ͯ͘
   • ٕज़తʹ͖Ε͍ʹͳ͚ͬͨͩͷچγεςϜ͸ͭ͘Βͳ͍
   

   View Slide

9. CDC࠾༻ͷഎܠ
   ϦϞσϦϯάΛ΍Δ
   • Ϣʔβʔͷߦಈ΍ಈػΛ෼ੳͯ͠
   ௚ײతͳυϝΠϯʢ༻ޠɾUIʣͱͯ͠࠶ఆٛ͢Δ
   • ϦϞσϦϯάͨ͠ϞσϧΛѻ͏৽γεςϜΛఏڙ͢Δ
   

   View Slide

10. CDC࠾༻ͷഎܠ
    Ϟσϧ͕ίϯςΩετΛ·͙ͨ
    • ҰํͰɺҰਓͷΞΫλʔ͔Βݟͨ௚ײతͳυϝΠϯ͸ɺผͷΞΫλʔ͔ΒݟΔͱ௚ײత͡Ό
    ͳ͍ࣄ͕͋Δ
    • ʮΞΫλʔ͕มΘΔͱؔ৺͕มΘΔʯΛແࢹͯ͠৽γεςϜ಺Ͱѻ͍ͬͯ͘ͱ·ͨಉ͡໰୊
    ͕ൃੜ͢Δ
    • ͏·͘ίϯςΩετΛ·͍ͨͰ͍͘࢓૊Έ͕ඞཁ
    => ΠϕϯτۦಈܕΞʔΩςΫνϟΛ࠾༻
    • ྫɿ ੡଄ύʔτφʔ͕"঎඼Λొ࿥"ͨ࣌͠ɺͦͷ঎඼͸αϙʔτνʔϜͷ
    "৹ࠪ଴ͪ঎඼"ϦετʹฒͿ
    

    View Slide

11. CDC࠾༻ͷഎܠ·Ͱ΋͏গ͠Ͱ͢
    

    View Slide

12. CDC࠾༻ͷഎܠ
    ΠϕϯτۦಈܕΞʔΩςΫνϟ
    • ΞϓϦέʔγϣϯΛߏ੒͢Δίϯϙʔωϯτ͕Πϕϯτʹج͍ͮͯಈ࡞͢Δઃܭύλʔϯ
    • ಛఆͷΠϕϯτ͕ൃੜͨ࣌͠ʹద੾ʹΞΫγϣϯΛى͜͢ࣄΛ໨త
    • × ঎඼Λొ࿥ͯ͠ʂ
    • ̋ ঎඼͕ొ࿥͞Ε·ͨ͠ʂ
    • εέʔϥϏϦςΟ, ॊೈੑ͕ߴ·Δ
    • Apache Kafka ͱ͍͏ϛυϧ΢ΣΞͷγΣΞ͕ଟ͍
    

    View Slide

13. CDC࠾༻ͷഎܠ
    Apache Ka)aΛ༻͍ͨ
    ΠϕϯτۦಈܕΞʔΩςΫ
    νϟ
    • ੡଄ύʔτφʔ༻ͷΞϓϦέʔγϣϯ
    • SoEʢྑ͍UXΛఏڙ͍ͨ͠!ʣ
    • ࣾ಺޲͚؅ཧը໘ͷΞϓϦέʔγϣϯ
    • SoRʢ͍ͭ୭͕ͲΜͳࢦࣔͷجͰ࣮ߦ͔ͨ͠. ཤྺʹ
    ॏ͖Λஔ͘ʣ
    • ΞΫλʔ΍ΞϓϦέʔγϣϯͷੑ࣭ͷҧ͍Λka-aΛ෗
    ഊ๷ࢭ૚ͱͯ͠औΓೖΕΔࣄͰંΓ߹͍͚ͭͯΔ
    

    View Slide

14. ҰํͰ..!
    

    View Slide

15. ແࢹͰ͖ͳ͍چγεςϜ
    • ݱঢ়΋Ϗδωε͸ಈ͍͍ͯͯͯɺಉ࣌ӡ༻͸ආ͚ΒΕͳ͍
    • ઌఔઃܭͨ͠ka$a΁ͷϝοηʔδͷૹ৴ΛچγεςϜʹରͯ͠
    खΛೖΕΔࣄ͸೉͍͠
    • ͳΜͱ͔ɺچγεςϜͱ͏·͘࿈ܞΛऔΔࣄ͕ग़དྷͳ͍͔ʁ
    

    View Slide

16. CDCΛ࠾༻͢Δͧʂʂ
    

    View Slide

17. CDC ͱ͸
    

    View Slide

18. CDCͱ͸
    • Change Data Capture
    • σʔλϕʔεͷมߋʢInsert,Update,DeleteʣΠϕϯτΛݕ஌͠
    ผͷΞϓϦέʔγϣϯʹ఻ୡ͢ΔͨΊͷٕज़΍ख๏
    • ࠾༻ࣄྫ
    • ৯΂ϩά͞Μɿ Elas5cSearch IndexͷϦΞϧλΠϜߋ৽
    • Chatwork͞Μɿ CQRSϞσϧͷߋ৽(ಡΈࠐΈϞσϧͷߋ৽)
    • ΦϛΧϨ͞Μɿ σʔλϕʔεϦϑΝΫλϦϯά
    • Debezium ౕ͕ͬͯྑ͍Β͍͠
    

    View Slide

19. Debeziumͷಛ௃
    

    View Slide

20. Debezium ͷಛ௃
    • Apache Ka*a Connect ʹ४ڌͯ͠࡞ΒΕͨ CDCͷπʔϧηοτ
    • Debeziumࣗମ͸ DBͷมߋΠϕϯτΛKa*aͷϝοηʔδͱͯ͠
    ൃՐ͢Δ੹຿
    • DB͝ͱʹConnectorΛ࢖ͬͯมߋΠϕϯτͷݕ஌Λ࣮ݱ͢Δ
    • MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, Cloud
    Spanner, ...
    

    View Slide

21. ࢖༻ྫ
    ͜ΜͳJSONΛ࡞Δ.
    {
    "name": "connector໊",
    "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "ུ": "ུ",
    "database.hostname": "{mysql}",
    "database.port": "{port}",
    "database.user": "{user}",
    "database.password": "{password}",
    "database.include.list": "{schema}",
    "table.include.list": "{schema.table1,...}",
    "snapshot.mode": "initial"
    }
    }
    ͜Μͳσʔλ͕ಘΒΕΔ
    // Insert
    {
    "before": null,
    "after": { "id": 1, "name": "init name" }
    }
    // Update
    {
    "before": { "id": 1, "name": "before name" },
    "after": { "id": 1, "name": "after name" }
    }
    // Delete
    {
    "before": { "id": 1, "name": "after name" },
    "after": null
    }
    

    View Slide

22. ӡ༻ʹ͓͍ͯؾΛ͚ͭΔࣄ
    Debezium
    • ConnectorͰಡΈࠐΉςʔϒϧΛ௥Ճ͢Δ࣌͸
    ৬ਓ͕ܳඞཁ
    ʢΫισΧίωΫλʔ໊Λ͚ͭͳ͍ʂ 1ഊʣ
    MySQL
    • ίωΫλఆٛ࣌ͷσϑΥϧτͷڍಈͱͯ͠
    ςʔϒϧͷશϨίʔυΛಡΈऔΔʢεφοϓγϣοτʣ
    • ಡऔதʹಉ͡Ϩίʔυ͕ಡΊΔΑ͏ʹϩοΫऔΔ
    ʢάϩʔόϧϦʔυϩοΫ or ςʔϒϧϩοΫʣ
    • ΊͪΌͪ͘ΌͰ͔͍ςʔϒϧΛಡΉ࣌͸΍͹͍ʢ΍͹͍ 1ഊʣ
    PostgreSQL
    • ಡऔઐ༻γϦΞϥΠβϒϧ෼཭Ϩϕϧͷ
    τϥϯβΫγϣϯΛషΔ
    • ϩοΫ͸ແ͍
    

    View Slide

23. ӡ༻ʹ͓͍ͯؾΛ͚ͭΔࣄͷվળྫ
    • DebeziumͷϩοΫΛؾʹ͢Δ৔߹͸ɺ
    CDCઐ༻ͷReadReplicaΛཱͯΔ
    • Debeziumࣗମʹࢮ׆؂ࢹΛೖΕ͓ͯ͘ͷ͕ྑ͍
    • αʔόʔ΍֤λεΫͷੜ͖ࢮʹ
    • ʮ๻͸ࡴ͞Ε·ͨ͠ɻࣗಈͰ͸ىಈग़དྷͳ͍ͷͰखಈͰىͯ͜͠ʯ
    ͬͯϩάʹग़ΔͷͰͦΕͷ؂ࢹ
    • ↑ ։ൃɾݕূ؀ڥͳͲͷ DBΛࢭΊΔ؀ڥͰൃੜ͢Δ
    • Subscrip6on Filter Ͱݕ஌ͯ͠ LambdaͰ Restart API Λࣗಈ࣮ߦ͢Δ
    

    View Slide

24. (10ϲ݄)ӡ༻ͯ͠ΈͯಘΒΕͨ஌ݟɾײ૝
    • ΊͪΌͪ͘Ό଎͍
    • APIʹϦΫΤετ౤͛ͯ
    Ϧϩʔυͨ͠Β൓ө͞ΕͯΔ
    • RESTͰͷઃఆָ͕
    • IaaC
    • ެࣜυΩϡϝϯτͷ໢ཏੑ
    • ϋϚͬͯ΋ɺجຊެࣜDoc͕Ҿ༻
    ͞ΕͯΔ
    • ϑϧϚωʔδυ͡Όͳ͍
    • MSKͰka'a-connectαϙʔτͯ͠
    Δ͚Ͳɺӡ༻ʹͷͤΔͷ͕ϦεΫ
    • ઃఆϛεͬͯΔͱͣͬͱϦτϥΠ
    ʢREST࢖͑ͳ͍ʣ
    • ϩʔΧϧ؀ڥ
    • DBͱͷ઀ଓ͕Ұఆࢭ·Δ࣌ʹ૬
    ੑѱΊ
    

    View Slide

25. Ҏ্Ͱ͢ʂ
    

    View Slide