[PATCH v4 7/9] powerpc/pseries/vas: Reopen windows with DLPAR core add

[Haren Myneni]

On Wed, 2022-02-23 at 17:28 +1000, Nicholas Piggin wrote:
> Excerpts from Haren Myneni's message of February 20, 2022 6:01 am:
> > VAS windows can be closed in the hypervisor due to lost credits
> > when the core is removed and the kernel gets fault for NX
> > requests on these in-active windows. If these credits are
> > available later for core add, reopen these windows and set them
> > active. When the OS sees page faults on these active windows,
> > it creates mapping on the new paste address. Then the user space
> > can continue to use these windows and send HW compression
> > requests to NX successfully.
> 
> Just for my own ignorance, what happens if userspace does not get
> another page fault on that window? Presumably when it gets a page
> fault it changes to an available window and doesn't just keep
> re-trying. So in what situation does it attempt to re-access a
> faulting window?

We should except faults only when the user space issue NX request on
these inactive windows. 
Example: 
- window is closed in the hypervisor to the DLPAR core removal. Means
inactive in the OS.
- window is opened later when the previous credit is available - means
active window in the OS.
- If the user space issue copy/paste on this window and fault on paste
address:
  - gets paste failure if the window is inactive
  - map to the new paste address if the window is active and returns 

So in case if the user space does not retry, new mapping will not be
done. From the OS point of view, this window is not used by the user
space. 

Thanks
Haren

> 
> Thanks,
> Nick
> 
> > Signed-off-by: Haren Myneni <haren at linux.ibm.com>
> > ---
> >  arch/powerpc/platforms/pseries/vas.c | 91
> > +++++++++++++++++++++++++++-
> >  1 file changed, 90 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/powerpc/platforms/pseries/vas.c
> > b/arch/powerpc/platforms/pseries/vas.c
> > index a297720bcdae..96178dd58adf 100644
> > --- a/arch/powerpc/platforms/pseries/vas.c
> > +++ b/arch/powerpc/platforms/pseries/vas.c
> > @@ -565,6 +565,88 @@ static int __init get_vas_capabilities(u8
> > feat, enum vas_cop_feat_type type,
> >  	return 0;
> >  }
> >  
> > +/*
> > + * VAS windows can be closed due to lost credits when the core is
> > + * removed. So reopen them if credits are available due to DLPAR
> > + * core add and set the window active status. When NX sees the
> > page
> > + * fault on the unmapped paste address, the kernel handles the
> > fault
> > + * by setting the remapping to new paste address if the window is
> > + * active.
> > + */
> > +static int reconfig_open_windows(struct vas_caps *vcaps, int
> > creds)
> > +{
> > +	long domain[PLPAR_HCALL9_BUFSIZE] = {VAS_DEFAULT_DOMAIN_ID};
> > +	struct vas_cop_feat_caps *caps = &vcaps->caps;
> > +	struct pseries_vas_window *win = NULL, *tmp;
> > +	int rc, mv_ents = 0;
> > +
> > +	/*
> > +	 * Nothing to do if there are no closed windows.
> > +	 */
> > +	if (!vcaps->nr_close_wins)
> > +		return 0;
> > +
> > +	/*
> > +	 * For the core removal, the hypervisor reduces the credits
> > +	 * assigned to the LPAR and the kernel closes VAS windows
> > +	 * in the hypervisor depends on reduced credits. The kernel
> > +	 * uses LIFO (the last windows that are opened will be closed
> > +	 * first) and expects to open in the same order when credits
> > +	 * are available.
> > +	 * For example, 40 windows are closed when the LPAR lost 2
> > cores
> > +	 * (dedicated). If 1 core is added, this LPAR can have 20 more
> > +	 * credits. It means the kernel can reopen 20 windows. So move
> > +	 * 20 entries in the VAS windows lost and reopen next 20
> > windows.
> > +	 */
> > +	if (vcaps->nr_close_wins > creds)
> > +		mv_ents = vcaps->nr_close_wins - creds;
> > +
> > +	list_for_each_entry_safe(win, tmp, &vcaps->list, win_list) {
> > +		if (!mv_ents)
> > +			break;
> > +
> > +		mv_ents--;
> > +	}
> > +
> > +	list_for_each_entry_safe_from(win, tmp, &vcaps->list, win_list)
> > {
> > +		/*
> > +		 * Nothing to do on this window if it is not closed
> > +		 * with VAS_WIN_NO_CRED_CLOSE
> > +		 */
> > +		if (!(win->vas_win.status & VAS_WIN_NO_CRED_CLOSE))
> > +			continue;
> > +
> > +		rc = allocate_setup_window(win, (u64 *)&domain[0],
> > +					   caps->win_type);
> > +		if (rc)
> > +			return rc;
> > +
> > +		rc = h_modify_vas_window(win);
> > +		if (rc)
> > +			goto out;
> > +
> > +		mutex_lock(&win->vas_win.task_ref.mmap_mutex);
> > +		/*
> > +		 * Set window status to active
> > +		 */
> > +		win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE;
> > +		mutex_unlock(&win->vas_win.task_ref.mmap_mutex);
> > +		win->win_type = caps->win_type;
> > +		if (!--vcaps->nr_close_wins)
> > +			break;
> > +	}
> > +
> > +	return 0;
> > +out:
> > +	/*
> > +	 * Window modify HCALL failed. So close the window to the
> > +	 * hypervisor and return.
> > +	 */
> > +	free_irq_setup(win);
> > +	h_deallocate_vas_window(win->vas_win.winid);
> > +	return rc;
> > +}
> > +
> >  /*
> >   * The hypervisor reduces the available credits if the LPAR lost
> > core. It
> >   * means the excessive windows should not be active and the user
> > space
> > @@ -673,7 +755,14 @@ static int vas_reconfig_capabilties(u8 type)
> >  	 * closed / reopened. Hold the vas_pseries_mutex so that the
> >  	 * the user space can not open new windows.
> >  	 */
> > -	if (old_nr_creds >  new_nr_creds) {
> > +	if (old_nr_creds <  new_nr_creds) {
> > +		/*
> > +		 * If the existing target credits is less than the new
> > +		 * target, reopen windows if they are closed due to
> > +		 * the previous DLPAR (core removal).
> > +		 */
> > +		rc = reconfig_open_windows(vcaps, new_nr_creds -
> > old_nr_creds);
> > +	} else {
> >  		/*
> >  		 * # active windows is more than new LPAR available
> >  		 * credits. So close the excessive windows.
> > -- 
> > 2.27.0
> > 
> > 
> >