[horizon] Fetch resources in parallel

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[horizon] Fetch resources in parallel

Ivan Kolodyazhny
Hi team,

We all know that unfortunately Horizon's performance not good enough in some cases. That's why we try to fix it on both sides: client-side and server-side.

I would like to talk mostly about server-side now. On some views, we use 'futurist' library [1] to fetch resources from API's in parallel. I've filed a blueprint for this effort [2]. You can find some work in progress patches in the Gerrit. 

For now, we use futurist.ThreadPoolExecutor in Horizon to fetch resources from APIs. It requires some specific Apache configuration changes to allow WSGI app to create threads. It means, our code execution depends on Apache mod_wsgi configuration. 

To avoid this, we can use futurist.GreenThreadPoolExecutor. It uses eventlet green threads which can be used to speed-up I/O operations like 'call external API'.

I did very simple performance testing [3] with proposed patch [4] for volumes views. My tests are not ideal but you can see how it's going with green thread. I propose to switch to the futurist.GreenThreadPoolExecutor and add 'eventlet.monkey_patch()' call to the wsgi app for Horizon. It will speed up parallel API calls and make Horizon independent on Apache or other web server configuration.

I added this topic to the next weekly meeting agenda [5] so we can discuss it there too.



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [horizon] Fetch resources in parallel

Ivan Kolodyazhny
Just forgot to mention one other option: we can use celery [6] too but I didn't do any performance testing with it. This solution will be more complex.


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Oct 31, 2017 at 7:15 PM, Ivan Kolodyazhny <[hidden email]> wrote:
Hi team,

We all know that unfortunately Horizon's performance not good enough in some cases. That's why we try to fix it on both sides: client-side and server-side.

I would like to talk mostly about server-side now. On some views, we use 'futurist' library [1] to fetch resources from API's in parallel. I've filed a blueprint for this effort [2]. You can find some work in progress patches in the Gerrit. 

For now, we use futurist.ThreadPoolExecutor in Horizon to fetch resources from APIs. It requires some specific Apache configuration changes to allow WSGI app to create threads. It means, our code execution depends on Apache mod_wsgi configuration. 

To avoid this, we can use futurist.GreenThreadPoolExecutor. It uses eventlet green threads which can be used to speed-up I/O operations like 'call external API'.

I did very simple performance testing [3] with proposed patch [4] for volumes views. My tests are not ideal but you can see how it's going with green thread. I propose to switch to the futurist.GreenThreadPoolExecutor and add 'eventlet.monkey_patch()' call to the wsgi app for Horizon. It will speed up parallel API calls and make Horizon independent on Apache or other web server configuration.

I added this topic to the next weekly meeting agenda [5] so we can discuss it there too.




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [horizon] Fetch resources in parallel

Ivan Kolodyazhny
Hi Team,

Last week we had a hot discussion about this topic at the meeting [6]. We still don't have an agreement how to go forward with it. As was requested, I did some perf testing with a current approach [3]. TBH, these tests results don't show real data but look realistic. I used the same VM with the same code (just checked out to the previous commit of horizon). To get better results, we need to re-test all but I don't have enough time now:(. I hope, these results are good enoudh and we can make our final decision on this topic.



Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Oct 31, 2017 at 11:06 PM, Ivan Kolodyazhny <[hidden email]> wrote:
Just forgot to mention one other option: we can use celery [6] too but I didn't do any performance testing with it. This solution will be more complex.


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Oct 31, 2017 at 7:15 PM, Ivan Kolodyazhny <[hidden email]> wrote:
Hi team,

We all know that unfortunately Horizon's performance not good enough in some cases. That's why we try to fix it on both sides: client-side and server-side.

I would like to talk mostly about server-side now. On some views, we use 'futurist' library [1] to fetch resources from API's in parallel. I've filed a blueprint for this effort [2]. You can find some work in progress patches in the Gerrit. 

For now, we use futurist.ThreadPoolExecutor in Horizon to fetch resources from APIs. It requires some specific Apache configuration changes to allow WSGI app to create threads. It means, our code execution depends on Apache mod_wsgi configuration. 

To avoid this, we can use futurist.GreenThreadPoolExecutor. It uses eventlet green threads which can be used to speed-up I/O operations like 'call external API'.

I did very simple performance testing [3] with proposed patch [4] for volumes views. My tests are not ideal but you can see how it's going with green thread. I propose to switch to the futurist.GreenThreadPoolExecutor and add 'eventlet.monkey_patch()' call to the wsgi app for Horizon. It will speed up parallel API calls and make Horizon independent on Apache or other web server configuration.

I added this topic to the next weekly meeting agenda [5] so we can discuss it there too.





__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [horizon] Fetch resources in parallel

Ivan Kolodyazhny
Hi Team,

It's me again and I've got some new tests results [3]. I'd got the following test environment:
  • 3 controllers 
  • 2 computes
  • 1 node with horizon
  • 60 items per page
  • 60 instances launched with only 3 different images.

As always, any feedback is welcome



Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Dec 6, 2017 at 12:47 PM, Ivan Kolodyazhny <[hidden email]> wrote:
Hi Team,

Last week we had a hot discussion about this topic at the meeting [6]. We still don't have an agreement how to go forward with it. As was requested, I did some perf testing with a current approach [3]. TBH, these tests results don't show real data but look realistic. I used the same VM with the same code (just checked out to the previous commit of horizon). To get better results, we need to re-test all but I don't have enough time now:(. I hope, these results are good enoudh and we can make our final decision on this topic.



Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Oct 31, 2017 at 11:06 PM, Ivan Kolodyazhny <[hidden email]> wrote:
Just forgot to mention one other option: we can use celery [6] too but I didn't do any performance testing with it. This solution will be more complex.


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Oct 31, 2017 at 7:15 PM, Ivan Kolodyazhny <[hidden email]> wrote:
Hi team,

We all know that unfortunately Horizon's performance not good enough in some cases. That's why we try to fix it on both sides: client-side and server-side.

I would like to talk mostly about server-side now. On some views, we use 'futurist' library [1] to fetch resources from API's in parallel. I've filed a blueprint for this effort [2]. You can find some work in progress patches in the Gerrit. 

For now, we use futurist.ThreadPoolExecutor in Horizon to fetch resources from APIs. It requires some specific Apache configuration changes to allow WSGI app to create threads. It means, our code execution depends on Apache mod_wsgi configuration. 

To avoid this, we can use futurist.GreenThreadPoolExecutor. It uses eventlet green threads which can be used to speed-up I/O operations like 'call external API'.

I did very simple performance testing [3] with proposed patch [4] for volumes views. My tests are not ideal but you can see how it's going with green thread. I propose to switch to the futurist.GreenThreadPoolExecutor and add 'eventlet.monkey_patch()' call to the wsgi app for Horizon. It will speed up parallel API calls and make Horizon independent on Apache or other web server configuration.

I added this topic to the next weekly meeting agenda [5] so we can discuss it there too.






__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev