User Tools

Site Tools


blog:2018:1213_more_is_less

Lessons in OpenStack - Less is More

OpenStack Version - Newton

I am a greedy programmer. I like big beefy machines with multi-cores and hundreds of gigabytes of RAM. This is a post on how OpenStack taught me a lesson that sometimes less is indeed more. Interestingly, I was not aware that the powerful Unix utility "less" gets its name due to a similar reason.

Background - I was trying to run a stereo-vision algorithm based on semi-global matching. The input images were quite large - 8000 x 8000 pixels. Stereo-matching is quite memory intensive but the operations can be heavily parallelized using OpenMP. So of course, I wanted to use as many cores as I could get my hands on.

On a physical machine with 28 physical cores (56 if you count hyperthreading) and 125 GB of RAM, it ran to completion in 4-5 minutes on average. My greed had paid off! The next step was to put it on our RVL Cloud so that multiple users could run the program simultaneously. I knew for a fact that our collaborators were using the same program on a VM with 60GB of RAM and 36 VCPUs and getting similar run times.

Elated with my success, I then converted the physical node to an OpenStack compute node and launched a VM (running the same OS, program etc.) on it with 60GB of RAM and 40 VCPUs (similar to what my collaborators had). The run time was around 10 minutes. Still not 5 minutes, but virtualization does add some overhead and the VM did have fewer number of cores. I decided to launch a bigger VM with 125 GB of RAM and 56 VCPUs to make up for the overhead of virtualization. Imagine my consternation as I watched the run time crawl to a whopping 40 minutes!!!

Then followed a period where I alternated between deep introspection and hair-pulling and wondered if I had fallen through a crack in space-time into an alternative reality. I finally realized that in my greed for cores, I had forgotten that OpenStack runs some native processes on the host machine. Effectively my big VM and OpenStack were competing for the 56 VCPUs. More google search indicated that this is an actual issue and that is why CPU pinning exists. From the RedHat guide, “The exact configuration depends on the NUMA topology of your host system; however, you must reserve some CPU cores across all the NUMA nodes for host processes and let the rest of the CPU cores handle your guest virtual machine instances”.

In my case, my VM was the only VM on the physical node, so the lack of CPU pinning was not the issue. Rather it was the fact that both the big VM and OpenStack were heavily competing for the VCPUs. With the smaller VM (with 40 VCPUs), there were 16 free VCPUs which OpenStack could use.

Lesson learnt - Sometimes, it pays to be thrifty.

~Bharath Comandur

This page was last edited at 2020/04/03 11:49

Discussion

Liwovosa, 2023/09/09 06:23

Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post. QQSLOT https://boltposts.com/qqslot/

MikeThomas, 2024/01/10 01:24

German fun is a delightful blend of tradition, innovation, and contagious enthusiasm. From the lively Oktoberfest celebrations to the whimsical Christmas markets, Germans know how to infuse joy into their cultural events with female escorts http://themodelsworldwide.com. Whether it's the cheerful sounds of traditional folk music, the vibrant colors of carnival parades, or the infectious laughter at local beer gardens, there's a sense of community and merriment that defines German fun.

asd, 2024/02/23 15:32

I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing. bimi logo

Amile Bills, 2024/03/01 16:19

I haven’t any word to appreciate this post…..Really i am impressed from this post….the person who create this post it was a great human..thanks for shared this with us. https://edureviewer.com/blog/

asd, 2024/03/04 05:33

Hello, this weekend is good for me, since this time i am reading this enormous informative article here at my home.Electrician service

SEO Expert, 2024/03/04 07:41

Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! Invest in Dubai https://www.seasidedubai.com/invest-in-dubai

asd, 2024/03/05 06:58

I was surfing the Internet for information and came across your blog. I am impressed by the information you have on this blog. It shows how well you understand this subject. atas

asd, 2024/03/05 07:51

I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. elektriker hamburg

casper, 2024/03/05 08:06

Thank you for everything. i really enjoy what you said and is very impress. Great work <a href=“https://www.atas8my.com” class=“urlextern” target=“_tab” title=“https://www.atas8my.com” rel=“dofollow noopener”>atas casino</a>

kantar, 2024/03/06 05:54

Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post.https://pogrebneusluge-laktasi.com/

kantar, 2024/03/06 05:56

I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing.https://rentacar-topauto.com/

stojce, 2024/03/06 05:58

I was surfing the Internet for information and came across your blog. I am impressed by the information you have on this blog. It shows how well you understand this subject. <a href=“https://rentacarbanjaluka.net/”>Rent a car Banja Luka</a>

kantar, 2024/03/06 06:00

I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing.[url=https://rentacar-topauto.com/]Rent a car Banja Luka[/url]

kantar, 2024/03/06 06:07

I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. Prevoz pokojnika iz inostranstva

rentacar, 2024/03/06 06:09

I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. Rent a car Banja Luka

rentacar, 2024/03/06 06:10

Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post.Rent a car banja luka

rentacar, 2024/03/06 06:12

I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing. Rent a car banja luka

rentacar, 2024/03/06 06:13

Thank you for everything. i really enjoy what you said and is very impress. Great work Rent a car banja luka

webdizajn-s, 2024/03/06 06:15

I was surfing the Internet for information and came across your blog. I am impressed by the information you have on this blog. It shows how well you understand this subject. Web dizajn banja luka

webdizajn-s, 2024/03/06 06:16

I was surfing the Internet for information and came across your blog. I am impressed by the information you have on this blog. It shows how well you understand this subject. Web dizajn

webdizajn-s, 2024/03/06 06:22

I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. Web dizajn banja luka

webdizajn-s, 2024/03/06 06:23

I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. Web dizajn banja luka

webdizajn-s, 2024/03/06 06:23

I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. Web dizajn

graficki dizajn, 2024/03/06 06:25

I was surfing the Internet for information and came across your blog. I am impressed by the information you have on this blog. It shows how well you understand this subject. Graficki dizajn banja luka

graficki dizajn, 2024/03/06 06:25

I was surfing the Internet for information and came across your blog. I am impressed by the information you have on this blog. It shows how well you understand this subject. Graficki dizajn

klizne kapije, 2024/03/06 06:27

I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing.Klizne kapije Banja Luka

klizne kapije, 2024/03/06 06:27

I recently found many useful information in your website especially this blog page. Among the lots of comments on your articles. Thanks for sharing.Klizne kapije

asd, 2024/03/07 07:28

Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.liga lotus

Amile Bills, 2024/03/10 09:25

You made such an interesting piece to read, giving every subject enlightenment for us to gain knowledge. Thanks for sharing the such information with us to read this… LeadMaker Media

asd, 2024/03/10 09:43

Thanks for the blog loaded with so many information. Stopping by your blog helped me to get what I was looking for. Federal Tax Credits SETC

asd, 2024/03/10 14:58

I love the way you write and share your niche! Very interesting and different! Keep it coming! denzeitungen

Amile Bills, 2024/03/11 08:23

It proved to be Very helpful to me and I am sure to all the commentators here! garage door repair

asd, 2024/03/14 08:42

Custom modern European doors are not only stylish but also incredibly functional. I appreciate the attention to detail and the innovative features that enhance both form and function. Custom Modern European Doors

Calgary Painters, 2024/03/16 08:51

I came to this site with the introduction of a friend around me and I was very impressed when I found your writing. I’ll come back often after bookmarking!Calgary Painters

Calgary Painters, 2024/03/16 08:53

I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you! Calgary Painters

asd, 2024/03/17 11:11

Great survey, I'm sure you're getting a great response. find help in Boise

asd, 2024/03/17 11:24

Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking.treatment for troubled teenagers

SEO, 2024/03/27 04:22

It is especially decent, though look into the tips during this home address.[https://jamboisnofal.com criminal lawyer]

Enter your comment. Wiki syntax is allowed:
P S H K U
 
blog/2018/1213_more_is_less.txt · Last modified: 2020/04/03 11:49 by admin